Implementation evals: 69 eval-banana harness_judge design-conformance checks#42
Merged
Conversation
… checks plan/implementation_evals/ — acceptance checks that judge the final implementation against the binding design (D1-D61 + requirements): one YAML per check, self-contained instructions citing the binding sources, conjunctive conditions, binary verdicts. Areas: E0 documents (8), E1 blocks/chunks (2), E2 claims (9), E3 relations/observations (5), entity resolution/review (3), K plane (4), projections (7), retrieval (6), ops (6), code/boundary (3). All 53 validated against the eval-banana check schema (unique ids, harness_judge, schema v1). questions.md §5: register two inconsistencies found while authoring — the requirements temporal-split paragraph predates D43/D49 (#30), and the requirements E3 bullet omits observations (#31). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…16 checks (69 total) Codex (gpt-5.5) reviewed the initial 53-check set (codex_review.md, committed alongside). Applied in full: - Factual must-fixes: deletion check now matches normal-delete retention semantics (claims/evidence kept as audit history; only hard-forget scrubs); execution-classes check names E2 extraction as the deliberate volume-proportional exception (D25/D31); Selection ledger check matches schema (plain keeps not ledgered); bitemporal check carries the D43 no-cap rule for measurements. - Judgeability: split the omnibus checks (boundary D60 -> 3, retrieval envelope D49 -> 5, lineage/living D55 -> 2); reworded checks to demand concrete evidence (shared hydrator boundary, import-linter contracts, startup/CI family validator, targeted reuse tests). - Coverage: +9 checks — D1/D46 source-of-truth split, D16 scope views, D22 golden sets, D23 registry scale, D36 sub-worker chain, code tooling/migrations, D62 delivery artifacts + enforced architecture, D63 embedding port config. - Mechanics: missing decision tags added, cite paths fixed, README re-inventoried (69 checks, scope D1-D63). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…D64 (seed core) in the predicate check + README range Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
a51a98c to
878926d
Compare
…eview)
Every check now carries an explicit source-precedence rule in its
instructions (judges never see the README): the design docs are the
binding statement of the current system; decision entries are read
with their refinement/withdrawal annotations; on any apparent
disagreement the judge treats the designs (incl.
postgres_schema_design.md) as controlling and reports the discrepancy
rather than failing a design-conformant implementation. The 27 checks
that cited only decisions gain explicit design-doc citations. README
records the convention ("decisions for coverage, designs for truth").
Rationale: the decision log is append-only testimony refined by later
entries; authoring checks from raw decision prose risks failing a
correct implementation on superseded text — the exact hazard Codex's
review caught once (the deletion check, corrected from
postgres_schema_design §13.1).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…coverage, designs for truth) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Author review round 2: the precedence rule alone left the checks decision-first in reading order (64/69 opened with "Read decisions.md Dxx and <design>"). Flipped all of them: the Read instruction now leads with the design doc / requirements section as "the binding source", with decisions.md cited second for rationale and refinement history. Also repaired ret_envelope_contradiction_completeness, whose description had been corrupted by the earlier cite-insertion (its description contained ", then", which the insertion script matched instead of the Read sentence). Full sweep: 69 checks valid, none decision-first, no polluted descriptions, every check cites a design/requirements source and carries the precedence rule. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hecks (76 total)
The round-2 review (codex_review_2.md, committed alongside) judged the
suite against the design docs as ground truth. All load-bearing quotes
were verified against the designs before applying. Applied in full:
- Surgery repairs: six mis-pathed requirements citations fixed
(plan/requirements/ prefix) + binding-source phrasing added to the
unflipped Read sentences; four non-design "binding source"
designations rebound (CLAUDE.md, concepts.md, workers.md, the P3
analysis are now context, not binding).
- Over-binding corrections (checks could fail conforming code):
provider calls through D61 ports are conformant; direct-env access
allowed via the requirement's reasoned per-line exception; ingest
bumps legitimately cascade downstream; effective_period added to the
claim_valid_kind enum; borderline per-claim grounding escalation is
designed; observations use the shared adjudication_outcome enum;
re-occurring relations in non-overlapping windows get new rows;
reviewer may be a designated agent; writers run parallel across
disjoint pages ("editing shared files" restored); model page binds
writer invocations, not every page; version-vs-lineage tombstone
branches spelled out; execution classes need an inspectable mapping
(no invented enum); retry bound is a tunable per-stage default;
producer/checker never-share per orchestration §8 (D53's
recorded-exception prose loses under precedence); projection purity
judged as outcome, not an invented import graph.
- D64 made scoreable (16-predicate seed incl. uses/reports_to
signatures); D23 expanded to the full partitioned estate + the
ungated load-test hook (a round-1 application miss).
- +7 coverage checks: K trigger surface/acyclicity, K eval canaries +
citation faithfulness, K writer runtime two-band, K hard-forget git
history, orchestration topology, S58 skill acceptance, continuous ER
health.
- Tags reconciled with scored conditions; README recounted (76),
column relabeled "primary binding sources", both reviews recorded.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
plan/implementation_evals/— the design-conformance acceptance suite for the final implementation: 69 eval-bananaharness_judgechecks, one YAML per check, each pointing an LLM judge at the binding sources (decision numbers + design docs) and at the code, with conjunctive conditions and a binary verdict. Discovered by eval-banana fromeval_checks/; run withuvx eval-banana run --harness-agent ….Coverage: D1–D63 + the requirements, across E0 documents (10), E1 blocks/chunks (2), E2 claims (9), E3 relations/observations (5), registries/ER/eval (5), K plane (4), projections (8), retrieval (10), ops/cross-cutting (8), code & boundary (8). Withdrawn decisions (D26–D30) and pure naming decisions excluded by design.
Conventions: absence = failure (acceptance for the complete system — filter by tags to match delivered phases); judge on a different model family than the producer (D53 applied to the eval itself); deliberate exceptions stated in-check so a correct implementation can't fail for obeying the design.
Codex review — applied in full
Per the process, Codex (gpt-5.5) independently reviewed the initial 53-check set (
codex_review.md, included). It caught 4 factual errors — most notably my deletion check contradicted the binding schema (normal delete retains claims/evidence as audit history; only hard-forget scrubs) and my execution-classes check wrongly demanded ambiguity-scaled spend from E2 extraction (deliberately volume-proportional per D25/D31). All must-fixes applied; omnibus checks split for judge reliability (boundary→3, envelope→5, lineage→2); 9 checks added for its coverage gaps (D1/D46, D16, D22, D23, D36, code tooling, D62×2, D63); tags/cites fixed. All 69 validated against the eval-banana check schema.Inconsistencies found while authoring (as requested)
Registered in
questions.md§5:(Also checked and found properly annotated: D37-vs-D51 raw-mount reversal, D12-vs-D45 hot-file removal, overall_design's K section.)
🤖 Generated with Claude Code